Stop trusting compromised eMudhra roots in the exploration-tests image by eliottness · Pull Request #13026 · DataDog/dd-trace-java

eliottness · 2026-10-23T17:30:37Z

What Does This Do

Stops the debugger exploration-tests image from trusting emSign Root CA - G1/G2, emSign ECC Root CA - G3, and AAA Certificate Services. dd-java-agent/agent-debugger/exploration-tests/Dockerfile.exploration-tests (base debian:bookworm-slim):

upgrades ca-certificates and re-runs update-ca-certificates -f;
updates the image JDK 17 to Temurin 17.0.21.0+9 and preinstalls Temurin 11.0.32.1+1; constrains Gradle to the preinstalled SDKMAN JDKs with toolchain auto-download disabled;
deletes all four roots from each installed SDKMAN JDK cacerts before Maven/Gradle download build inputs;
sanitizes affected entries in the google.p12 and google.jks stores embedded in cached google-api-client jars after the OkHttp Gradle warm-up;
deletes Gradle's generated gradle-api-*.jar files, which embed an old CA snapshot.

The APPSEC-71273 ledger lists 17 impacted image references in this repository: 0 fixed here, 1 fixed at their in-repo image source (clean once that image is republished), 16 follow-ups.

Motivation

Incident IR-62343: the affected eMudhra/AAA trust anchors in Java include emSign Root CA - G1/G2, emSign ECC Root CA - G3, and AAA Certificate Services. The JDKs still include some of these roots even in the newest available Temurin 11/17 builds, so the image must remove them before dependency resolution.

Additional Notes

The exploration-tests jobs run on the prebuilt registry.ddbuild.io/ci/dd-trace-java:exploration-tests image, so this change takes effect only after someone runs the manual build-exploration-tests-image job once it is merged. That tag has no clean published version yet.
An earlier run of exploration-tests-method-jsoup failed on this PR. The failure cannot come from this Dockerfile, because the job used the unchanged prebuilt image; touching dd-java-agent/agent-debugger/** only made the job run. Other PRs touching the debugger passed the same job that day (Require explicit locales and charsets in tracer code #13020 and Close tracers built by debugger tests (quick fix) #13040), so this PR was rebased onto master to re-run the pipeline.
The earlier image scan covered the previous Dockerfile revision. This review-follow-up revision has not been rebuilt or scanned locally; the manual image-build job must build and scan this version before the prebuilt tag is republished. Images that CI never builds or pulls are left alone.

Follow-ups (not in this PR):

Needs a rebuild at its source (Datadog-built or mirrored image) (13): registry.ddbuild.io/images/mirror/amazon/aws-cli:2.4.29, 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/benchmarking-platform:java-dsm-kafka, 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/benchmarking-platform:java-debugger, registry.ddbuild.io/images/mirror/dd-apm-test-agent/ddapm-test-agent:v1.64.1, registry.ddbuild.io/images/mirror/datadog/agent:7.40.1, registry.ddbuild.io/images/mirror/sqlserver:latest, registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-base, registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-7, registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-8, registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-11, registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-17, registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-21, registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-25.
No clean upstream version yet (2): gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators, mcr.microsoft.com/mssql/server:latest.
Unscannable from outside the registry (pull access denied) (1): 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/agent-key-management-tools/gpg:1.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: APPSEC-71277 (parent APPSEC-71273)

datadog-datadog-prod-us1 · 2026-10-23T17:47:54Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 73.97% (+14.46%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 8c6efd9 | Docs | Give us feedback!}

dd-octo-sts · 2026-10-23T17:58:30Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.05 s	13.97 s	[-0.4%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent	12.95 s	12.98 s	[-0.9%; +0.4%] (no difference)
startup:petclinic:appsec:Agent	17.72 s	17.58 s	[-0.1%; +1.8%] (no difference)
startup:petclinic:iast:Agent	17.55 s	17.61 s	[-1.0%; +0.4%] (no difference)
startup:petclinic:profiling:Agent	17.24 s	17.40 s	[-1.8%; -0.1%] (maybe better)
startup:petclinic:sca:Agent	17.51 s	16.86 s	[-0.6%; +8.2%] (no difference)
startup:petclinic:tracing:Agent	16.21 s	16.79 s	[-7.6%; +0.6%] (no difference)

Commit: 8c6efd97 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Upgrade ca-certificates in the debian:bookworm-slim base and delete emSign ECC Root CA - G3 / AAA Certificate Services from every SDKMAN and Gradle-provisioned JDK cacerts in the exploration-tests image (IR-62343). The image is published by the manual build-exploration-tests-image job; re-run it after merge. Jira: APPSEC-71277

pr-commenter · 2026-10-26T13:03:07Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1793211374	1793211701
end_time	2026-10-28T18:17:42	2026-10-28T18:23:08
git_branch	master	eliottness/emudhra-ca-roots
git_commit_sha	`ab5833d`	`8c6efd9`
start_time	2026-10-28T18:16:15	2026-10-28T18:21:42

See matching parameters

	Baseline	Candidate
ci_job_id	2187314671	2187314671
ci_pipeline_id	148303879	148303879
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1793210752	1793210752

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 6 unstable metrics.

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-31.029µs; +15.622µs] or [-9.733%; +4.900%]	unstable [-46.508µs; +23.596µs] or [-13.130%; +6.661%]	unstable [-58.017µs; +34.624µs] or [-15.704%; +9.372%]	unstable [-63.463µs; +134.085µs] or [-5.155%; +10.892%]	same
scenario:basic	same	same	same	unstable [-10.700µs; +249.153µs] or [-0.941%; +21.919%]	unstable [-64.348op/s; +170.056op/s] or [-2.831%; +7.482%]
scenario:loop	same	same	same	same	same

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (354.219 µs) : 314, 394
.   : milestone, 354,
basic (329.222 µs) : 323, 335
.   : milestone, 329,
loop (8.096 ms) : 8033, 8159
.   : milestone, 8096,
section candidate
noprobe (342.763 µs) : 318, 368
.   : milestone, 343,
basic (326.277 µs) : 321, 332
.   : milestone, 326,
loop (8.107 ms) : 8034, 8181
.   : milestone, 8107,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	354.219 µs [314.256 µs, 394.183 µs]
basic	329.222 µs [323.482 µs, 334.962 µs]
loop	8.096 ms [8.033 ms, 8.159 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	342.763 µs [317.834 µs, 367.693 µs]
basic	326.277 µs [320.872 µs, 331.682 µs]
loop	8.107 ms [8.034 ms, 8.181 ms]

chatgpt-codex-connector · 2026-10-26T13:33:47Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-26T13:40:49.968800Z	`e253125`	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-26T13:37:01.440873Z	`e253125`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e25312578b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

datadog-datadog-prod-us1

Deleting Gradle's generated API jar leaves AAA Certificate Services in its source google.p12 truststore. Gradle can continue trusting that root independently of JDK cacerts and regenerate the removed snapshot.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit e253125}

Preinstall Java 11 and explicitly constrain Gradle toolchain paths so the OkHttp job cannot create an unscrubbed JDK at runtime. Scrub all four affected roots before resolving Maven and Gradle dependencies, and remove them from cached Google PKCS12 and JKS truststores after warm-up. Co-authored-by: sisyphus-dev-ai <sisyphus-dev-ai@users.noreply.github.com>

Addresses review: the Java 11 and 17 SDKMAN versions are declared once as ARGs and reused.

jpbempel · 2026-10-28T12:00:16Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-10-28T12:00:21Z

View all feedbacks in Devflow UI.

2026-10-28 12:00:21 UTC ℹ️ Start processing command /merge

2026-10-28 12:00:25 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-10-28 12:43:21 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 576f565:

exploration-tests-metric_line-jackson-core

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

eliottness · 2026-10-28T15:36:35Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-10-28T15:36:39Z

View all feedbacks in Devflow UI.

2026-10-28 15:36:39 UTC ℹ️ Start processing command /merge

2026-10-28 15:36:44 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-10-28 16:15:25 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 7790d03:

Ubuntu_20_amd64.SI94: [test-app-java-alpine]

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

eliottness · 2026-10-28T16:17:46Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-10-28T16:17:50Z

View all feedbacks in Devflow UI.

2026-10-28 16:17:50 UTC ℹ️ Start processing command /merge

2026-10-28 16:17:54 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-10-28 16:43:45 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for f66a452:

exploration-tests-log_method-okhttp

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

eliottness · 2026-10-28T23:31:58Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-10-28T23:32:03Z

View all feedbacks in Devflow UI.

2026-10-28 23:32:02 UTC ℹ️ Start processing command /merge

2026-10-28 23:32:08 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).

2026-10-29 00:16:53 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 4ee80f1:

exploration-tests-metric_line-jackson-core

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

eliottness force-pushed the eliottness/emudhra-ca-roots branch from 626140e to e253125 Compare October 26, 2026 12:41

eliottness added tag: no release notes Changes to exclude from release notes comp: testing Testing labels Oct 26, 2026

eliottness marked this pull request as ready for review October 26, 2026 13:33

eliottness requested a review from a team as a code owner October 26, 2026 13:33

eliottness requested review from ojung and removed request for a team October 26, 2026 13:33

chatgpt-codex-connector Bot reviewed Oct 26, 2026

View reviewed changes

datadog-datadog-prod-us1 Bot reviewed Oct 26, 2026

View reviewed changes

Comment thread dd-java-agent/agent-debugger/exploration-tests/Dockerfile.exploration-tests

jpbempel requested review from jpbempel and removed request for ojung October 26, 2026 16:40

jpbempel approved these changes Oct 28, 2026

View reviewed changes

Comment thread dd-java-agent/agent-debugger/exploration-tests/Dockerfile.exploration-tests Outdated

Make the exploration-test JDK versions build args

75ecc77

Addresses review: the Java 11 and 17 SDKMAN versions are declared once as ARGs and reused.

Merge branch 'master' into eliottness/emudhra-ca-roots

8c6efd9

Conversation

eliottness commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-datadog-prod-us1 Bot commented Oct 23, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

pr-commenter Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

chatgpt-codex-connector Bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jpbempel commented Oct 28, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What to do next?

Uh oh!

eliottness commented Oct 28, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What to do next?

Uh oh!

eliottness commented Oct 28, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What to do next?

Uh oh!

eliottness commented Oct 28, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What to do next?

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

eliottness commented Oct 23, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Oct 23, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

dd-octo-sts Bot commented Oct 23, 2026 •

edited

Loading

pr-commenter Bot commented Oct 26, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 26, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 28, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 28, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 28, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 28, 2026 •

edited

Loading